Image processing device, method, and program

ABSTRACT

The present invention relates to an image processing apparatus and a method and a program that enable an amount of code to be suppressed below a desired value with more reliability. An emergency mode determining section  142  determines that the amount of code of a next picture is likely to exceed an upper limit if a target bit calculated by a rate controlling section  118  exceeds a predetermined upper limit. If it is determined that the amount of code of the next picture is likely to exceed the upper limit, the code amount controlling section  142  controls an orthogonal transducing section  114  to replace a transform coefficient of a predetermined frequency component with zero, controls an intra predicting section  125  to restrict intra prediction modes to be applied, and controls a mode determining section  127  to restrict inter prediction modes to be applied. The prevent invention can be applied to an encoding apparatus.

TECHNICAL FIELD

The present invention relates to an image processing apparatus and amethod and a program, and in particular, relates to an image processingapparatus and a method and a program that enable an amount of code to besuppressed below a desired value with more reliability.

BACKGROUND ART

In a system that encodes an image in accordance with an H.264/AVC(Advanced Video Coding) system, an upper limit of an amount of code perpicture is set in accordance with conditions such as an HRD(Hypothetical Reference Decoder) buffer capacity, a network bandwidth, asystem layer constraint, and a readout performance of a disk drive. Ifan amount of code exceeds the upper limit, operation of a system maymalfunction. Accordingly, there have conventionally been proposedvarious methods for controlling an amount of code in an encodingapparatus that encodes an image in accordance with an H.264/AVC system(e.g., see Patent Document 1).

Patent Document 1: Japanese Patent Application Laid-Open 2004-165894

DISCLOSURE OF INVENTION Technical Problem

However, for example, in the case where an image including white noiseis input, there is a possibility that an amount of code exceeds an upperlimit by conventional control based on a rate control method defined inan MPEG-2 Test Model 5 (TM5).

The present invention has been made in view of the above situation, andenables an amount of code to be suppressed below a desired value withmore reliability.

Technical Solution

An image processing apparatus according to one aspect of the invention,in the image processing apparatus for encoding an image in accordancewith an MPEG (Moving Picture Coding Experts Group) 4 system or anH.264/AVC (Advanced Video Coding) system, includes determination meansfor determining whether or not an amount of code per picture controlledby first code amount control means for controlling a bit rate is likelyto exceed a predetermined upper limit, and second code amount controlmeans for controlling encoding to suppress the amount of code inaccordance with a method different from that of the first code amountcontrol means if it is determined that the amount of code per picture islikely to exceed the upper limit.

The determination means can determine whether or not the amount of codeper picture is likely to exceed the predetermined upper limit, based onan amount of code allocated to each picture by the first code amountcontrol means.

The second code amount control means can perform control to replace,with zero, a transform coefficient of a predetermined frequencycomponent among transform coefficients obtained by performing anorthogonal transform on a macroblock in a difference image which is adifference between an original image and a prediction image.

The second code amount control means can perform control to replace atransform coefficient of a frequency component excluding adirect-current component with zero.

The second code amount control means can perform control so as toreplace transform coefficients of all frequency components including adirect-current component with zero.

The second code amount control means can perform control to change afrequency component range in which a transform coefficient is replacedwith zero, based on an amount of code allocated to each picture by thefirst code amount control means.

The second code amount control means can perform control to change afrequency component range in which a transform coefficient is replacedwith zero, based on a type of a slice to which a macroblock belongs.

The second code amount control means can control a prediction mode whichis applied to generate a prediction image.

The second code amount control means can perform control for applying asame prediction direction to each block which is a unit for performingan intra prediction.

The second code amount control means can perform control for applying aDC mode to each block.

The second code amount control means can perform control for applying anintra 16×16 mode, in an intra prediction of a luminance signal.

The second code amount control means can control a size of a block whichis a unit for performing the intra prediction, based on the type of theslice to which a macroblock belongs.

The second code amount control means can perform control for applying askip mode as a prediction mode for a macroblock belonging to a P slice.

The second code amount control means can perform control for applying askip mode or a direct mode as a prediction mode for a macroblockbelonging to a B slice.

An image processing method according to one aspect of the invention, inthe image processing method for encoding an image in accordance with anMPEG (Moving Picture Coding Experts Group) 4 system or an H.264/AVC(Advanced Video Coding) system, includes the steps of determiningwhether or not an amount of code per picture controlled by code amountcontrol means for controlling a bit rate is likely to exceed apredetermined upper limit, and controlling encoding to suppress theamount of code in accordance with a method different from that of thecode amount control means if it is determined that the amount of codeper picture is likely to exceed the upper limit.

A program according to one aspect of the invention, in the program forallowing a computer to execute image processing for encoding an image inaccordance with an MPEG (Moving Picture Coding Experts Group) 4 systemor an H.264/AVC (Advanced Video Coding) system, includes the steps ofdetermining whether or not an amount of code per picture controlled bycode amount control means for controlling a bit rate is likely to exceeda predetermined upper limit, and controlling encoding to suppress theamount of code in accordance with a method different from that of thecode amount control means if it is determined that the amount of codeper picture is likely to exceed the upper limit.

According to one aspect of the invention, it is determined whether ornot an amount of code per picture controlled by code amount controlmeans for controlling a bit rate is likely to exceed a predeterminedupper limit, and encoding is controlled to suppress the amount of codein accordance with a method different from that of the code amountcontrol means if it is determined that the amount of code per picture islikely to exceed the upper limit.

ADVANTAGEOUS EFFECTS

According to one aspect of the invention, the amount of code issuppressed. Further, according to one aspect of the invention, theamount of code can be suppressed below a desired value with morereliability.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an embodiment of an image processingapparatus to which the invention is applied.

FIG. 2 is a flowchart for explaining encoding processing performed bythe image processing apparatus in FIG. 1.

FIG. 3 is a flowchart for explaining emergency mode dealing processingperformed by the image processing apparatus in FIG. 1.

FIG. 4 is a block diagram showing an example of the configuration of apersonal computer.

EXPLANATION OF REFERENCE NUMERALS

101—Image processing apparatus, 114—Orthogonal transducing section,115—Quantizing section, 118—Rate controlling section, 119—Emergency modecontrolling section, 125—Intra predicting section, 126—Motionpredicting/compensating section, 127—Mode determining section,141—Emergency mode determining section, 142—Code amount controllingsection

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, an embodiment of the present invention will be describedwith reference to the accompanying drawings.

FIG. 1 is a block diagram showing a configuration of an embodiment of animage processing apparatus to which the invention is applied.

An image processing apparatus 101 is an apparatus that encodes an inputimage in accordance with an H.264/AVC (Advanced Video Coding) system andoutputs the encoded image, for example, to a recording apparatus, atransmission line, or the like, which is arranged on a subsequent stage,not shown in drawings.

The image processing apparatus 101 is configured to include an A/D(Analog/Digital) converting section 111, a screen image rearrangementbuffer 112, an adder 113, an orthogonal transducing section 114, aquantizing section 115, a lossless encoding section 116, a storagebuffer 117, a rate controlling section 118, an emergency modecontrolling section 119, an inverse quantizing section 120, an inverseorthogonal transducing section 121, an adder 122, a deblocking filter123, a frame memory 124, an intra predicting section 125, a motionpredicting/compensating section 126, and a mode determining section 127.Further, the emergency mode controlling section 119 is configured toinclude an emergency mode determining section 141 and a code amountcontrolling section 142.

The A/D converting section 111 A/D-converts an analog image input froman outside into a digital image, and supplies the converted digitalimage (hereinafter also referred to as an original image as appropriate)to the screen image rearrangement buffer 112.

The screen image rearrangement buffer 112 rearranges original imagessupplied from the A/D converting section 111, based on a GOP (Group OfPictures) structure, and supplies them in order to the adder 113, theintra predicting section 125, and the motion predicting/compensatingsection 126.

The adder 113 acquires, from the mode determining section 127, either anintra prediction image predicted using an intra prediction (intra-frameprediction) on each macroblock in the original image or an interprediction image predicted using an inter prediction (inter-frameprediction, motion-compensating prediction). The adder 113 finds adifference for each macroblock between the original image and the intraprediction image or the inter prediction image, and supplies adifference image composed of prediction errors obtained by finding thedifference to the orthogonal transducing section 114.

The orthogonal transducing section 114 performs, on each block of apredetermined size in the difference image, an orthogonal transform suchas a discrete cosine transform or a Karhunen-Loeve transform, andsupplies a transform coefficient thereby obtained to the quantizingsection 115. Further, as will be described later with reference to FIG.3, in an emergency mode in which an amount of code per picture is likelyto exceed a predetermined upper limit, the orthogonal transducingsection 114 replaces a value of the transform coefficient of apredetermined frequency component with zero, under the control of theemergency mode section 126.

The quantizing section 115 quantizes the transform coefficient suppliedfrom the orthogonal transducing section 114, using a quantization scalecontrolled by the rate controlling section 118, and supplies thequantized transform coefficient to the lossless encoding section 116 andthe inverse quantizing section 120.

The lossless encoding section 116 acquires information on the intraprediction from the intra predicting section 125, and acquiresinformation on the inter prediction from the motionpredicting/compensating section 126. The lossless encoding section 117arranges the quantized transform coefficient, the information on theintra prediction, the information on inter prediction, etc. in apredetermined order, and performs, on the arranged data, losslessencoding processing including variable length coding such as CAVLC(Context-Adaptive Variable Length Coding), arithmetic coding such asCABAC (Context-Adaptive Binary Arithmetic Coding) or the like. Thelossless encoding section 116 supplies the encoded data to the storagebuffer 117 for storage.

The storage buffer 117 outputs the data, as an image encoded with theH.264/AVC system, supplied from the lossless encoding section 116 to arecording apparatus, a transmission line, or the like, which is arrangedon the subsequent stage, not shown in drawings.

Based on the amount of code of the image stored in the storage buffer117, the rate controlling section 118 controls a bit rate which is anamount of code per time allocated to an image to be encoded.

For example, the rate controlling section 118 controls a bit rate bycontrolling a value of a quantization scale being a value by which thequantizing section 115 divides the transform coefficient inquantization, using a rate control method defined in an MPEG-2 TestModel 5 (TM5). Further, the rate controlling section 118 calculates atarget bit which is an estimate value of an amount of code allocated toan image (picture) to be encoded next, and supplies informationindicating the calculated target bit to the emergency mode controllingsection 119.

The emergency mode controlling section 119 performs processingconcerning an emergency mode.

The emergency mode determining section 141 among elements constitutingthe emergency mode controlling section 119 determines whether or not anamount of code per picture of an image (picture) to be encoded next islikely to exceed a predetermined upper limit, based on the target bitacquired from the rate controlling section 118. The emergency modedetermining section 141 supplies information indicating thedetermination result to the code amount controlling section 142.

Note that the upper limit of the amount of code per picture is set basedon conditions such as an HRD buffer capacity, a network bandwidth, asystem layer constraint, and a readout performance of a disk drive.Further, the upper limit may be made variable and varied according to achange in conditions.

If the emergency mode determining section 141 determines that the amountof code per picture is likely to exceed the predetermined upper limit,the code amount controlling section 142 controls encoding to suppressthe amount of code by controlling processing performed by the orthogonaltransducing section 114, the intra predicting section 125, or the modedetermining section 127 in accordance with a method different from thatof the rate controlling section 118, as will be described later withreference to FIG. 3.

The inverse quantizing section 120 inverse-quantizes the transformcoefficient supplied from the quantizing section 115, and supplies it tothe inverse orthogonal transducing section 121.

The inverse orthogonal transducing section 121 performs, on thetransform coefficient supplied from the inverse quantizing section 120,an inverse orthogonal transform such as an inverse discrete cosinetransform and an inverse Karhunen-Loeve transform. Thereby, thedifference image is decoded. The inverse orthogonal transducing section121 supplies the decoded difference image to the adder 122.

The adder 122 acquires from the mode determining section 127 the intraprediction image or the inter prediction image which is used to generatethe difference image, and adds the difference image to the acquiredintra prediction image or inter prediction image. Thereby, the originalimage is decoded. The adder 122 supplies the decoded image (hereinafterreferred to as the decoded image as appropriate) to the deblockingfilter 123.

The deblocking filter 123 performs, on the decoded image, deblockingprocessing for eliminating block distortion. The deblocking filter 123supplies the image in which the deblocking processing is performed tothe frame memory 124. Further, the deblocking filter 123 supplies thedecoded image as it is without the deblocking processing as an image foruse in the intra prediction to the frame memory 124.

The frame memory 124 stores the image supplied from the deblockingfilter 123, as an image (hereinafter referred to as a reference image asappropriate) which is referred to in performing the intra prediction orthe inter prediction.

The intra predicting section 125 performs the intra prediction, on eachmacroblock, for generating an intra prediction image for the originalimage, using encoded pixels adjacent to a macroblock in the same framestored in the frame memory 124. Note that as described above, the pixelsof the decoded image before the deblocking processing is performed areused in the intra prediction.

The motion predicting/compensating section 126 detects a motion vectorfor each macroblock in the original image with respect to the referenceimage, using the reference image of another frame stored in the framememory 124, and motion-compensation is performed to the reference image,using the detected motion vector, thereby performing the interprediction for generating the inter prediction image for the originalimage.

The mode determining section 127 determines a prediction mode to beapplied to each macroblock, using for example, a Low Complexity Mode(high-speed mode).

Now, more specific description will be made of processing by the intrapredicting section 125, the motion predicting/compensating section 126,and the mode determining section 127.

In the H.264/AVC standard, a plurality of prediction modes are definedas an intra prediction method and an inter prediction method.

Specifically, as for the intra prediction of luminance signals, threekinds which are 4×4 pixels (intra 4×4 mode), pixels (intra 8×8 mode,only in the case where High Profile is applied), and 16×16 pixels (intra16×16 mode) are defined as units of blocks for the intra prediction.

Further, in the intra 4×4 mode and the intra 8×8 mode, nine kinds ofprediction modes 0 to 8 are defined as pixel value predictiondirections. In the intra 16×16 mode, four kinds of prediction modes 0 to3 are defined as pixel value prediction directions.

Further, as for the intra prediction of color difference signals, onekind which is 8×8 pixels is defined as a unit of a block for the intraprediction, and four kinds of prediction modes 0 to 3 are defined aspixel value prediction directions.

Moreover, as for the inter prediction, seven kinds which are 16×16pixels, 16×8 pixels, 8×16 pixels, 8×8 pixels, 8×4 pixels, 4×8 pixels,and 4×4 pixels are defined as units of blocks for inter prediction.Further, as for the inter prediction of blocks belonging to a B(Bi-directional Predictive) slice, there are defined a previousprediction mode in which only previous (past) pictures are referred toon a temporal axis, a subsequent prediction mode in which onlysubsequent (future) pictures are referred to on the temporal axis, abidirectional prediction mode in which both the previous and subsequentpictures are referred to on the temporal axis, and the like.

Moreover, as for the inter prediction of blocks belonging to a P(Predictive) slice or the B slice, there is defined a skip mode forsuppressing the amount of code of information on the macroblock byusing, as it is, the pixel value of the block of the reference imagereferred to by the motion vector. Further, as for the inter predictionof blocks belonging to the B slice, there is defined a direct mode forpredicting the motion vector by using the motion vector of an encodedblock.

The intra predicting section 125 calculates, for each macroblock, costvalues Cost(Mode), by a following equation (1), concerning all theprediction modes of the intra prediction (hereinafter also referred toas intra prediction modes) which are applicable to each macroblock.

Cost(Mode)SA(T)D+SA(T)DO  (1)

Here, SA(T)D denotes an absolute sum of prediction errors which are thedifference between the original image and the prediction image. Also,SA(T)DO is an offset value given to the SA(T)D and is an amount of dataused for the transmission of header information added to the macroblocksuch as the motion vector.

The intra predicting section 125 supplies to the mode determiningsection 127, all the calculated cost values Cost(Mode), that is,information indicating the cost values Cost(Mode) for all the intraprediction modes which are applicable to the macroblocks.

Further, as will be described later with reference to FIG. 3, in anemergency mode, the intra predicting section 125 restricts theprediction modes for which the cost values Cost(Mode) are calculated,under the control of the code amount controlling section 119, thusrestricting the intra prediction modes to be applied to each macroblock.

As with the intra predicting section 125, the motionpredicting/compensating section 126 calculates, for each macroblock, thecost values Cost(Mode), by the above equation (1), concerning all theprediction modes of the inter prediction (hereinafter also referred toas inter prediction modes) which are applicable to each macroblock. Themotion predicting/compensating section 126 supplies to the modedetermining section 127, all the calculated cost values Cost(Mode), thatis, information indicating the cost values Cost(Mode) for all the interprediction modes which are applicable to the macroblocks.

The mode determining section 127 selects the prediction modes to beapplied to each macroblock from the viewpoint of image quality and theamount of code, based on the cost values Cost(Mode) acquired from theintra predicting section 125 or the motion predicting/compensatingsection 126.

Further, as will be described later with reference to FIG. 3, in anemergency mode, the mode determining section 127 restricts theprediction modes to be applied to each macroblock, under the control ofthe code amount controlling section 119.

If the mode determining section 127 selects one of the intra predictionmodes, the mode determining section 127 supplies information indicatingthe selected intra prediction mode to the intra predicting section 125.The intra predicting section 125 generates an intra prediction imagebased on the selected intra prediction mode, and supplies the generatedintra prediction image to the mode determining section 127. Further, theintra predicting section 125 supplies information such as the selectedprediction mode as information on the intra prediction of anintra-predicted macroblock, to the lossless encoding section 116.

Further, if the mode determining section 127 selects one of the interprediction modes, the mode determining section 127 supplies informationindicating the selected inter prediction mode to the motionpredicting/compensating section 126. The motion predicting/compensatingsection 126 detects the motion vector and generates the inter predictionimage based on the selected inter prediction mode, and supplies thegenerated inter prediction image to the mode determining section 127.Further, the motion predicting/compensating section 126 suppliesinformation such as the selected prediction mode, the detected motionvector, and the number of the reference image (picture) as informationon the inter prediction of the inter-predicted macroblock, to thelossless encoding section 116.

The mode determining section 127 supplies the intra prediction image orthe inter prediction image generated based on the selected predictionmode to the adder 113 and the adder 122.

Next, referring to a flowchart in FIG. 2, description will be made ofencoding processing performed by the image processing apparatus 101 inFIG. 1. Note that this processing is started, for example, when an imageinput from an outside to the image processing apparatus 101 is started.

In Step S1, the image processing apparatus 101 starts image encoding.That is, the sections in the image processing apparatus 101 start theoperations described with reference to FIG. 1, thereby starting toencode the input image in accordance with the H.264/AVC system. Further,emergency mode dealing processing which will be described later withreference to FIG. 3 is started.

In Step S2, the image processing apparatus 101 determines whether allthe images have been encoded. In Step S2, image encoding is performeduntil it is determined that all the images input from the outside havebeen encoded. If it is determined that all the images input from theoutside have been encoded, the encoding processing ends.

Next, referring to a flowchart in FIG. 3, description will be made ofemergency mode dealing processing performed by the image processingapparatus 101 during the encoding processing described above withreference to FIG. 2.

In Step S21, the rate controlling section 118 calculates the target bitof the next picture. Specifically, the rate controlling section 118calculates the target bit of the next picture, based on the amount ofcode of the image stored in the storage buffer 117, for example, using arate control method defined in the TM5. The rate controlling section 118supplies information indicating the calculated target bit to theemergency mode determining section 141.

In Step S22, the emergency mode determining section 141 determineswhether or not the amount of code of the next picture is likely toexceed an upper limit. The emergency mode determining section 141determines that the amount of code of the next picture is likely toexceed the upper limit if the target bit acquired from the ratecontrolling section 118 exceeds a predetermined upper limit, and theprocessing proceeds to Step S23.

In Step S23, the code amount controlling section 142 providesinstructions concerning emergency mode operations. Specifically, theemergency mode determining section 141 supplies information indicatingthat the amount of code of the next picture is likely to exceed theupper limit to the code amount controlling section 142. The code amountcontrolling section 142 provides instructions concerning emergency modeoperations to the orthogonal transducing section 114, the intrapredicting section 125, and the mode determining section 127.

In Step S24, the orthogonal transducing section 114 starts to replacethe transform coefficient of a predetermined frequency component withzero, under the control of the code amount controlling section 142.

For example, in the case where T denotes the target bit of the nextframe, Θ denotes the upper limit of the amount of code per picture, andTh1, Th2, and Th3 (in this regard, Th1<Th2<Th3) denote predeterminedthreshold values regarding the amount of code per picture, under thecontrol of the code amount controlling section 142, if Th1<T−Θ<Th2, theorthogonal transducing section 114 replaces the transform coefficientvalues of a predetermined frequency component excluding a direct-current(DC) component and another specific frequency component other than itwith zero. If Th2≦T−®<Th3, the orthogonal transducing section 114replaces the transform coefficient values of a frequency componentexcluding the DC component with zero. If Th3≧T−Θ, the orthogonaltransducing section 114 replaces the transform coefficient values of allthe frequency components including the DC component with zero. That is,a frequency component range in which the transform coefficient isreplaced with zero is changed based on the target bit T. Morespecifically, the frequency component range in which the transformcoefficient is replaced with zero is increased so that the amount ofcode is suppressed more as the target bit T increases.

Further, the frequency component range in which the transformcoefficient is replaced with zero may be changed, for example, based onthe type of a slice to which a macroblock belongs. For example, as for amacroblock belonging to an I (Intra) slice, the transform coefficientvalues of all the frequency components including the DC component may bereplaced with zero, and as for a macroblock belonging to the P slice orthe B slice, the transform coefficient values of the frequency componentexcluding the DC component may be replaced with zero. That is, thefrequency component range in which the transform coefficient is replacedwith zero is increased so that the amount of code is suppressed more ina macroblock belonging to the I slice with the amount of code tending tobe larger than that of the P or B slice than in a macroblock belongingto the P or B slice.

Thus, the transform coefficient of the predetermined frequency componentis replaced with zero, thereby suppressing the amount of code.

Note that the orthogonal transducing section 114 continues thereplacement of the transform coefficient with zero until the emergencymode is canceled in Step S28 described later.

In Step S25, the intra predicting section 125 restricts intra predictionmodes to be applied, under the control of the code amount controllingsection 142. For example, under the control of the code amountcontrolling section 142, the intra predicting section 125 limits thepixel value prediction directions of the intra 4×4 mode, the intra 8×8mode (only in the case where the High Profile is applied), and the intra16×16 mode of luminance signals and color difference signalsrespectively to one kind. The intra predicting section 125 calculatesonly a cost value Cost(Mode) for the limited prediction direction, andstops calculating cost values Cost(Mode) for the other predictiondirections. Thereby, in the intra prediction mode, the prediction modesof all the same prediction direction are applied. As a result, itbecomes unnecessary to encode the prediction mode of each block, thusmaking it possible to suppress the amount of code.

Note that in this case, the prediction directions are unified into a DCmode (a prediction mode 2 in the luminance signals and a prediction mode0 in the color difference signals), thereby unnatural directivity doesnot appear in an image, compared with a case where they are unified intoanother prediction mode. Thus image degradation can be suppressed.

Further, for example, as for the luminance signals, the cost valueCost(Mode) may be calculated only for the intra 16×16 mode in which asmaller amount of code occurs.

Moreover, the prediction modes to be applied may be restricted, forexample, based on the type of a slice to which a macroblock belongs. Forexample, as for luminance signals of a macroblock belonging to the Islice, the cost value Cost(Mode) may be calculated only for the intra16×16 mode, and as for luminance signals of a macroblock belonging tothe P slice or the B slice, the cost values Cost(Mode) may be calculatedfor the intra 4×4 mode, the intra 8×8 mode (in the case of the HighProfile), and the intra 16×16 mode, respectively. That is, a block whichis a unit for performing the intra prediction is set to be larger sothat the amount of code is suppressed more in a macroblock belonging tothe I slice having the amount of code tending to be larger than that ofthe P or B slice than in a macroblock belonging to the P or B slice.

Thus, the intra prediction modes to be applied to each macroblock arerestricted so that the amount of code in intra prediction is suppressed.

Note that the intra predicting section 125 continues the restriction ofintra prediction modes until the emergency mode is canceled in Step S28described later.

In Step S26, the mode determining section 127 restricts the interprediction modes to be applied, under the control of the code amountcontrolling section 142.

For example, as for a macroblock belonging to the B slice, the modedetermining section 127 restricts the choices of the inter predictionmodes to a skip mode and a direct mode, under the control of the codeamount controlling section 142. That is, the prediction modes to beapplied to the macroblock belonging to the B slice are restricted to anyone of the intra prediction mode limited in Step S25, the direct mode,and the skip mode.

Further, for example, as for a macroblock belonging to the P slice, themode determining section 127 restricts the choices of the interprediction modes to the skip mode, under the control of the code amountcontrolling section 142. That is, the prediction modes to be applied tothe macroblock belonging to the P slice are restricted to either of theintra prediction mode limited in Step S25 and the skip mode.

Furthermore, as for a macroblock belonging to the I slice, since theinter prediction is not performed, the prediction modes to be appliedare restricted to the intra prediction mode limited in Step S25.

Based on the cost value Cost(Mode), the mode determining section 127selects a prediction mode to be applied to the macroblock, from amongthe restricted prediction modes.

Thereby, in the case where the inter prediction is selected, since theprediction mode is made to serve as the skip mode or the direct mode,the amount of code is suppressed.

Further, as for a macroblock belonging to the P or B slice, the intraprediction mode may be removed from the choices. Thereby, the skip modeof the inter prediction is applied to the macroblock belonging to the Pslice, and the direct mode or the skip mode of the inter prediction isapplied to the macroblock belonging to the B slice, so that the amountof code can be suppressed more.

Note that the mode determining section 127 continues the restriction ofthe inter prediction modes until the emergency mode is canceled in StepS28 described later.

After that, the processing returns to Step S21 to execute Step S21 andthe subsequent steps.

In Step S22, if it is determined that the amount of code of the nextpicture is not likely to exceed the upper limit, the processing proceedsto Step S27.

In Step S27, the code amount controlling section 142 determines whetheror not it is in the emergency mode. If it is determined that it is inthe emergency mode, the processing proceeds to Step S28.

In Step S28, the code amount controlling section 142 cancels theemergency mode. Specifically, the code amount controlling section 142instructs the orthogonal transducing section 114, the intra predictingsection 125, and the mode determining section 127 to cancel theemergency mode. Thereby, the orthogonal transducing section 114, theintra predicting section 125, and the mode determining section 127 stopthe emergency mode operations and resume regular operations.

After that, the processing returns to Step S21 to execute Step S21 andthe subsequent steps.

In Step S27, if it is determined that it is not in the emergency mode,the processing returns to Step S21 to execute Step S21 and thesubsequent steps.

As described above, if it is determined that the amount of code perpicture is likely to exceed the predetermined upper limit, the transformcoefficient of the predetermined frequency component is replaced withzero, the intra prediction modes to be applied are restricted, or theinter prediction modes to be applied are restricted, thereby making itpossible to suppress the amount of code below the upper limit with morereliability.

Further, only in the emergency mode, the transform coefficient of thepredetermined frequency component is replaced with zero, the intraprediction modes to be applied are restricted, and the inter predictionmodes to be applied are restricted. Therefore, unnecessary imagedegradation is prevented.

Note that in the above explanation, examples are set that allprocessings of Steps S24 to S26 are executed in the emergency mode.However, only processing of one or two steps may be executed.

Further, while processes to be executed in respective steps of Steps S24to S26 are selected based on the target bit value, the type of a sliceto which a macroblock belongs, or the like as described above, the stepto be actually executed may be selected from among Steps S24 to S26. Forexample, in the case where Th11, Th12, and Th13 (in this regard,Th11<Th12<Th13) denote predetermined threshold values regarding theamount of code per picture, if Th11<T−Θ<Th12, the intra prediction modesmay be restricted to the DC mode in Step S25. If Th2≦T−Θ<Th3, the intraprediction modes may be restricted to the intra 16×16 mode at Step S25.If Th3≧T-Θ, the transform coefficient of the predetermined frequencycomponent may be replaced with zero in Step S24.

Moreover, the determination processing in Step S22 in FIG. 3 is notlimited to the above example, and may be performed based on a valueother than the target bit.

Furthermore, in the above explanation, encoding is performed inaccordance with the H.264/AVC system, however, the present invention isalso applicable to the case where encoding is performed in accordancewith any other encoding method such as the MPEG-4 (Moving Picture CodingExperts Group phase 4).

The above-described consecutive processings can be executed not only byhardware but also by software. In the case of using software to executethe consecutive processings, a program forming the software is installedfrom a program recording medium onto a computer incorporated indedicated hardware, or e.g. a general-purpose personal computer that canexecute various functions by installing various programs thereon.

FIG. 4 is a block diagram showing an example of the configuration of apersonal computer 300 which performs the above-described consecutiveprocessings in accordance with programs. A CPU (Central Processing Unit)301 performs various processings in accordance with programs stored in aROM (Read Only Memory) 302 or a recording section 308. In a RAM (RandomAccess Memory) 303, programs performed by the CPU 301, data, and thelike are stored as appropriate. The CPU 301, the ROM 302, and the RAM303 are interconnected via a bus 304.

An input-output interface 305 is connected to the CPU via the bus 304.An input section 306 composed of a keyboard, a mouse, a microphone, andthe like and an output section 307 composed of a display, a speaker, andthe like are connected to the input-output interface 305. The CPU 301performs various processings in accordance with an instruction inputfrom the input section 306. Then, the CPU 301 outputs processing resultsto the output section 307.

The recording section 308 connected to the input-output interface 305 iscomposed of, e.g., a hard disk, and stores programs performed by the CPU301 and various kinds of data. A communication section 309 communicateswith an external apparatus via a network such as the Internet or a localarea network.

Alternatively, a program may be acquired through the communicationsection 309 and stored in the recording section 308.

When removable media 311 such as a magnetic disk, an optical disk, amagneto-optical disk, or a semiconductor memory are mounted on a drive310 connected to the input-output interface 305, the drive 310 drivesthem and acquires a program and data stored therein. The acquiredprogram and data are transferred and stored into the recording section308 as necessary.

As shown in FIG. 4, a program recording medium for storing programswhich are installed in the computer and made executable by the computerincludes the removable media 311 which are package media composed of themagnetic disk (including a flexible disk), the optical disk (including aCD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital VersatileDisc)), the magneto-optical disk, the semiconductor memory, or the like;or the ROM 302 in which the programs are stored temporarily orpermanently; and the hard disk constituting the recording section 308.The programs are stored in the program recording medium through the useof a wired or wireless communication medium such as the local areanetwork, the Internet, or digital satellite broadcasting through thecommunication section 309 which is an interface such as a router or amodem as necessary.

Further, the steps used, in the present specification, to describe theprograms to be stored in the program recording medium represent not onlyprocessing to be performed in a time-series manner in the order they arementioned but also processing to be performed, not necessarily in atime-series manner, concurrently or individually.

Moreover, the embodiment of the present invention is not limited to theabove embodiment, and various modifications can be made withoutdeparting from the scope and spirit of the invention.

1. An image processing apparatus for encoding an image in accordancewith an MPEG (Moving Picture Coding Experts Group) 4 system or anH.264/AVC (Advanced Video Coding) system, comprising: determinationmeans for determining whether or not an amount of code per picturecontrolled by first code amount control means for controlling a bit rateis likely to exceed a predetermined upper limit; and second code amountcontrol means for controlling encoding to suppress the amount of code inaccordance with a method different from that of the first code amountcontrol means if it is determined that the amount of code per picture islikely to exceed the upper limit.
 2. The image processing apparatus asset forth in claim 1, wherein the determination means determines whetheror not the amount of code per picture is likely to exceed thepredetermined upper limit, based on an amount of code allocated to eachpicture by the first code amount control means.
 3. The image processingapparatus as set forth in claim 1, wherein the second code amountcontrol means performs control to replace, with zero, a transformcoefficient of a predetermined frequency component among transformcoefficients obtained by performing an orthogonal transform on amacroblock in a difference image which is a difference between anoriginal image and a prediction image.
 4. The image processing apparatusas set forth in claim 3, wherein the second code amount control meansperforms control to replace a transform coefficient of a frequencycomponent excluding a direct-current component with zero.
 5. The imageprocessing apparatus as set forth in claim 3, wherein the second codeamount control means performs control to replace transform coefficientsof all frequency components including a direct-current component withzero.
 6. The image processing apparatus as set forth in claim 3, whereinthe second code amount control means performs control to change afrequency component range in which a transform coefficient is replacedwith zero, based on an amount of code allocated to each picture by thefirst code amount control means.
 7. The image processing apparatus asset forth in claim 3, wherein the second code amount control meansperforms control to change a frequency component range in which atransform coefficient is replaced with zero, based on a type of a sliceto which a macroblock belongs.
 8. The image processing apparatus as setforth in claim 1, wherein the second code amount control means controlsa prediction mode which is applied to generate a prediction image. 9.The image processing apparatus as set forth in claim 8, wherein thesecond code amount control means performs control for applying a sameprediction direction to each block which is a unit for performing anintra prediction.
 10. The image processing apparatus as set forth inclaim 9, wherein the second code amount control means performs controlfor applying a DC mode to each block.
 11. The image processing apparatusas set forth in claim 8, wherein the second code amount control meansperforms control for applying an intra 16×16 mode in intra prediction ofa luminance signal.
 12. The image processing apparatus as set forth inclaim 8, wherein the second code amount control means controls the sizeof a block which is a unit for performing an intra prediction, based ona type of a slice to which a macroblock belongs.
 13. The imageprocessing apparatus as set forth in claim 8, wherein the second codeamount control means performs control for applying a skip mode as aprediction mode for a macroblock belonging to a P slice.
 14. The imageprocessing apparatus as set forth in claim 8, wherein the second codeamount control means performs control for applying a skip mode or adirect mode as a prediction mode for a macroblock belonging to a Bslice.
 15. An image processing method for encoding an image inaccordance with an MPEG (Moving Picture Coding Experts Group) 4 systemor an H.264/AVC (Advanced Video Coding) system, comprising the steps of:determining whether or not an amount of code per picture controlled bycode amount control means for controlling a bit rate is likely to exceeda predetermined upper limit; and controlling encoding to suppress theamount of code in accordance with a method different from that of thecode amount control means if it is determined that the amount of codeper picture is likely to exceed the upper limit.
 16. A program forallowing a computer to execute image processing for encoding an image inaccordance with an MPEG (Moving Picture Coding Experts Group) 4 systemor an H.264/AVC (Advanced Video Coding) system, comprising the steps of:determining whether or not an amount of code per picture controlled bycode amount control means for controlling a bit rate is likely to exceeda predetermined upper limit; and controlling encoding to suppress theamount of code in accordance with a method different from that of thecode amount control means if it is determined that the amount of codeper picture is likely to exceed the upper limit.